Improve Performance: CoPilot: users experience #110

quge009 · 2025-10-27T22:54:50Z

This PR is mainly about improving the user experience.

Changes made to optimize users perceived latency and reading experience:
- Implement streaming output for LLMSession class, change the final answer generation call to streaming output, to post the answer to user as-soon-as the first few tokens are ready.
- Implement the push_frontend method to leverage the steaming output to feedback the CoPilot progress status message to user in real-time, to manage users' experience during waiting for the answer.
- Add auto scroll feature for frontend plugin to enhance readability.
Changes made to reduce the average_response_latency (defined as time between question receival and answer posting):
- Refactor several components' (SmartHelp, LTP, ...) implementation into classes, to make it possible to preserve the states when necessary.
- Reuse the same llm_session instance for requests within the same conversation, by avoiding unnecessary https re-connection in initialization.
- Implement a new question parsing function to combine contextualization and classification llm calls into one efficient call, to reduce time.
- Move prompt reading to instance initialization, to avoid unnecessary file I/O operations.
Also a minor bug fix is included:
- Change the assignment of 'turnId' to frontend.

Effectiveness of this PR:

Impact on accuracy
- No change
Impact on response latency
- ~15% response time reduction on average
- ~50% response time reduction for extreme simple question

Copilot

Pull Request Overview

This pull request refactors the CoPilot chat agent to improve scalability and add streaming support. The main changes include:

Refactoring global singletons to instance-based LLM sessions: Removes global LLMSession() instances and passes them as parameters to avoid blocking in multi-user scenarios
Adding streaming support: Implements Server-Sent Events (SSE) streaming for real-time response delivery to the frontend
Improving thread safety: Adds locks for authentication state and introduces per-instance stream callbacks

Reviewed Changes

Copilot reviewed 26 out of 28 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/copilot-chat/src/copilot_agent/utils/llmsession.py	Added streaming methods, per-instance callbacks, config caching, and thread safety improvements
src/copilot-chat/src/copilot_agent/utils/summary.py	Updated to accept `llm_session` parameter instead of using global instance
src/copilot-chat/src/copilot_agent/utils/smart_help.py	Refactored from function to class-based `SmartHelp` for better state management
src/copilot-chat/src/copilot_agent/ltp/ltp.py	Converted from module-level functions to `LTP` class with instance-based session handling
src/copilot-chat/src/copilot_agent/copilot_service.py	Added streaming endpoint and session management per user/conversation
src/copilot-chat/src/copilot_agent/copilot_conversation.py	Updated to use per-conversation LLM sessions
src/copilot-chat/src/copilot_agent/utils/push_frontend.py	New module for pushing events to frontend via streaming
contrib/copilot-plugin/src/app/ChatBox.tsx	Frontend updated to consume SSE streaming responses
contrib/copilot-plugin/src/app/ChatHistory.tsx	Enhanced auto-scroll behavior for streaming updates

Files not reviewed (1)

contrib/copilot-plugin/package-lock.json: Language not supported

Comments suppressed due to low confidence (3)

src/copilot-chat/src/copilot_agent/copilot_conversation.py:205

This comment appears to contain commented-out code.

        # try:
        #     push_frontend_meta(response_message_info)
        # except Exception:
        #     logger.debug('Failed to push early meta event for streaming client')

src/copilot-chat/src/copilot_agent/utils/dcw.py:123

Call to function gen_dcw with too few arguments; should be no fewer than 3.

    full_dcw = gen_dcw(user_prompt, map_existing)

src/copilot-chat/src/copilot_agent/copilot_turn.py:56

This statement is unreachable.

            question = this_inquiry

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/copilot-chat/src/copilot_agent/utils/summary.py

contrib/copilot-plugin/src/app/ChatBox.tsx

src/copilot-chat/src/copilot_agent/copilot_conversation.py

src/copilot-chat/src/copilot_agent/utils/logger.py

contrib/copilot-plugin/src/app/ChatHistory.tsx

src/copilot-chat/src/copilot_agent/copilot_turn.py

src/copilot-chat/src/copilot_agent/copilot_service.py

src/copilot-chat/src/copilot_agent/ltp/ltp.py

src/copilot-chat/src/copilot_agent/copilot_conversation.py

…nt route

Co-authored-by: Copilot <[email protected]>

…ation

…on results

…loop

…ion manager

…approach to detect incremental updates.

Co-authored-by: Copilot <[email protected]>

…sions dictionary

…extract_session_info(data).

…on to make sure each thread uses its session

hippogr

I left a single comment, which is minor for you to review and resolve it. Besides of that, I guess we are good to go. ✌️

quge009 temporarily deployed to auto-test October 27, 2025 22:54 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 27, 2025 23:44 — with GitHub Actions Inactive

quge009 had a problem deploying to auto-test October 28, 2025 00:22 — with GitHub Actions Failure

quge009 temporarily deployed to auto-test October 28, 2025 19:11 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 19:21 — with GitHub Actions Inactive

quge009 changed the title ~~tmp~~ Improve Performance: CoPilot, response latency, user expectation Oct 28, 2025

quge009 changed the title ~~Improve Performance: CoPilot, response latency, user expectation~~ Improve Performance: CoPilot: response latency, user expectation Oct 28, 2025

quge009 changed the title ~~Improve Performance: CoPilot: response latency, user expectation~~ Improve Performance: CoPilot: users' perceived response latency Oct 28, 2025

quge009 marked this pull request as ready for review October 28, 2025 20:04

quge009 requested review from Copilot, hippogr, yukirora and zhogu October 28, 2025 20:04

Copilot AI reviewed Oct 28, 2025

View reviewed changes

quge009 temporarily deployed to auto-test October 28, 2025 20:09 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:19 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:36 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:41 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:46 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:48 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:56 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 20:59 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 21:00 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test October 28, 2025 21:02 — with GitHub Actions Inactive

quge009 changed the title ~~Improve Performance: CoPilot: users' perceived response latency~~ Improve Performance: CoPilot: users experience Oct 28, 2025

quge009 temporarily deployed to auto-test October 28, 2025 21:46 — with GitHub Actions Inactive

quge009 had a problem deploying to auto-test October 28, 2025 22:18 — with GitHub Actions Failure

quge009 had a problem deploying to auto-test October 28, 2025 22:20 — with GitHub Actions Failure

quge009 had a problem deploying to auto-test October 28, 2025 22:24 — with GitHub Actions Failure

quge009 and others added 26 commits November 13, 2025 15:30

update: nginx configuration to add the new /copilot/api/stream endpoi…

d638e20

…nt route

Update src/copilot-chat/src/copilot_agent/copilot_conversation.py

02fe508

Co-authored-by: Copilot <[email protected]>

remove unnecessary comment

1ff7fba

Update src/copilot-chat/src/copilot_agent/copilot_turn.py

80f00e0

Co-authored-by: Copilot <[email protected]>

resolve review comment: remove consle log

3ee9f37

Update src/copilot-chat/src/copilot_agent/copilot_service.py

b0ec9e0

Co-authored-by: Copilot <[email protected]>

Update src/copilot-chat/src/copilot_agent/ltp/ltp.py

084ceb6

Co-authored-by: Copilot <[email protected]>

Update src/copilot-chat/src/copilot_agent/copilot_conversation.py

ce18d6a

Co-authored-by: Copilot <[email protected]>

update: remove unused function

b0f6cc4

improve: robustness, gracefully handling if classification fail

ad4d68a

change classifier version for deployment

6a73d70

resolve comment: change class name for clarity

56a9ab3

resolve comment: 1.handle exceptions when necessary, 2.improve logging

c89d90c

resolve comment: edit job metadata query to remove return count limit…

2515afd

…ation

resolve comment: rewrite logger module using logging instead of print

a82f63d

resolve comments: replace magic numbers with consts, for classificati…

6b85fd1

…on results

resolve comment: frontend plugin, add safety check to avoid infinite …

51884cc

…loop

fix bug: streaming output outputs the aggregated message to conversat…

8d63d8e

…ion manager

resolve copilot comment: frontend, robust comparison or length-based …

7f514d0

…approach to detect incremental updates.

Update src/copilot-chat/src/copilot_agent/utils/summary.py

9d3831e

Co-authored-by: Copilot <[email protected]>

resolve copilot comment: fix typo

f332ae7

Update src/copilot-chat/src/copilot_agent/copilot_service.py

99d5491

Co-authored-by: Copilot <[email protected]>

resolve copilot comment: add thread lock to protect access to the ses…

1293c81

…sions dictionary

resolve copilot comment: add developer notes

b32c10a

resolve copilot comment: extracting this logic into a helper method _…

6fdb523

…extract_session_info(data).

resolve copilot comment: add a explicitly method to reassign llmsessi…

558637e

…on to make sure each thread uses its session

quge009 force-pushed the lequ/copilot/improve-perf branch from d52405d to 558637e Compare November 13, 2025 23:30

quge009 temporarily deployed to auto-test November 13, 2025 23:31 — with GitHub Actions Inactive

quge009 temporarily deployed to auto-test November 13, 2025 23:35 — with GitHub Actions Inactive

hippogr approved these changes Nov 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve Performance: CoPilot: users experience #110

Improve Performance: CoPilot: users experience #110

quge009 commented Oct 27, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hippogr left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Improve Performance: CoPilot: users experience #110

Are you sure you want to change the base?

Improve Performance: CoPilot: users experience #110

Conversation

quge009 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR is mainly about improving the user experience.

Effectiveness of this PR:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hippogr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

quge009 commented Oct 27, 2025 •

edited

Loading